Scene Segmentation of 3D Kinect Images with Recursive Neural Networks

In this project, we study scene segmentation of images from the Microsoft Kinect using deep learning techniques. The Kinect gives a depth map of the scene in addition to a standard RGB image, so we are extending methods for scene segmentation and object recognition developed by Socher’s group which were previously applied to two-dimensional images. Socher’s algorithm parses scenes using recursive neural networks (RNNs) to discover recursive structure in the image.1 With the availability of a large standardized data set gathered from the Kinect by researchers at NYU,2 a natural step is to extend Sochers code to apply the RNN-based algorithm to the three-dimensional Kinect images. Our algorithm starts by oversegmenting the image based on RGB data with the Edge Detection and Image Segmentation (EDISON) system. We compute vision features in these superpixels and map these into a neural network which outputs a “semantic” feature representation for the superpixels. The vision features that we use involve both standard two-dimensional image features as well as features based on the depth information gathered from the Kinect, including measures of the distance and normal vector. The recursive neural network then computes a score indicating whether pairs of adjacent segments should be identified as part of a larger